When an image is gotten (i.e., captured) with a digital camera, noise may be added to the image depending on a characteristic of its own CCD (charge-coupled device) or CMOS read circuit or a property of the transmission line. In addition, the image may also get blurred due to either out-of-focus shooting or a camera shake. In this manner, the quality of an image shot may deteriorate due to not only the noise resulting from the image's own characteristic but also the blur that has been caused by hand during the shooting operation. Among these multiple different kinds of “blurs”, the blur of an image that has been induced by the movement of a camera during shooting (or exposure) will be sometimes referred to herein as a “motion blur”.
Recently, as there is a growing demand for high-sensitivity shooting, it has become more and more necessary to restore such a blurry deteriorated image (which will be simply referred to herein as a “deteriorated image”) to the original condition (which will be referred to herein as an “ideal image”) as perfectly as possible. To capture a bright and noise- or blur-free image that should be obtained by high-sensitivity shooting, there are roughly two approaches: one of them is to increase the sensitivity and the other is to extend the exposure time.
If the sensitivity is increased, however, then the noise will be amplified, too. As a result, the signal will be drowned in the noise, and the resultant image will be mostly covered with the noise in many cases. On the other hand, if the exposure time is extended, the light produced on the spot can be accumulated a lot and the resultant image will have much less noise. In that case, the signal will not be drowned in the noise but the image will get shaky much more easily due to a camera shake, which is a problem.
Thus, in order to extend the exposure time, two different approaches have been taken so far. One of them is optical image stabilization such as a lens shift or a sensor shift. According to the other method, the direction and/or the magnitude of the motion blur is/are obtained based on the captured image or by the sensor and then subjected to signal processing, thereby restoring the image (which will be referred to herein as a “restoration method by signal processing”).
The optical image stabilization has only a limited correction range. If the exposure time is extended, a camera shake will be produced more easily, and therefore, the lens or the sensor needs to be shifted in a broader range. However, the longer the distance the lens or sensor needs to go, the longer the time delay caused by their shift. In addition, a physical limit prevents the designer from increasing the size of the camera unlimitedly.
Meanwhile, the restoration methods by signal processing are disclosed in Patent Documents Nos. 1 and 2 and Non-Patent Documents Nos. 1 and 2, for example. Hereinafter, those restoration methods by signal processing will be described.
Suppose the intensity distribution of an image produced on the imaging area of an image sensor is represented by I(x, y). The coordinates (x, y) are two-dimensional coordinates indicating the location of a pixel (i.e., a photosensitive cell) on the imaging area. For example, if the image is made up of M×N pixels that are arranged in columns and rows and if x and y are integers that satisfy the inequalities 0≦x≦M−1 and 0≦y≦N−1, respectively, then the location of each of the pixels that form the image can be indicated by a set of coordinates (x, y). In this example, the origin (0, 0) of the coordinate system is supposed to be put at the upper left corner of the image, the x-axis is supposed to run vertically and the y-axis is supposed to run horizontally. However, the coordinate system may also be defined in any other way.
If the intensity distribution of a blur-free image (which is an ideal image or what the image should be) is represented by L(x, y), a point spread function (PSF) that defines the blur is represented by PSF(x, y) and the noise is represented by n(x, y), then the following Equation (1) is satisfied:I(x,y)=PSF(x,y)*L(x,y)+n(x,y)  (1)where the sign * indicates a two-dimensional convolution operation.
The point spread function PSF(x, y) of a camera shake depends on the trace of the camera that has been shaken (which will be referred to herein as a “camera shake trace”) during shooting (or exposure). Since the camera shake trace changes every time an image is shot with the camera, PSF(x, y) also changes every shooting session with the camera.
If the camera shake trace during shooting has been detected by a gyrosensor, for example, and if PSF(x, y) is known, the deteriorated image I(x, y) can be restored to an image L(x, y) by performing a deconvolution operation using that PSF(x, y). On the other hand, unless PSF(x, y) is known, the deteriorated image I(x, y) needs to be restored to the image L(x, y) by estimating PSF(x, y) based on the deteriorated image I(x, y). The former method is called a “non-blind deconvolution”, while the latter method is called a “blind deconvolution”. Since PSF(x, y) and the image L(x, y) both need to be estimated based on the deteriorated image I(x, y) according to the blind deconvolution method, it is more difficult to reduce the blur than when the non-blind deconvolution method is adopted.
A PSF convolution operation that defines a camera-shake-induced motion blur is performed with a linear filter. A two-dimensional convolution operation linear filter is usually represented by a kernel consisting of a coefficient matrix with a size of N×N pixels, where N is an integer that is equal to or greater than three. The PSF that defines a blur can be represented by a blur kernel. Thus, to restore a blurry image to a less blurred (or non-blurred) image, the blur kernel that defines the blur needs to be estimated.
As a method for restoring an image by signal processing, Non-Patent Document No. 1 discloses that a multi-scale inference scheme is used to infer a blur kernel and a restored image based on a single blurry image. According to that multi-scale inference scheme, first of all, a blur kernel having a size of 3×3 pixels is inferred by using a deteriorated image with a low resolution. After that, by gradually increasing the resolution of the deteriorated image for use in inference, the resolution of the blur kernel is also raised. FIGS. 19(a) through 19(h) illustrate the method disclosed in Non-Patent Document No. 1. The upper part of FIG. 19(a) shows a blur kernel of 3×3 pixels, while the lower part shows a restored image with a corresponding resolution. The same statement applies to FIGS. 19(b) through 19(h), in which the resolution increases gradually in this order.
As there are a small number of pixels in the deteriorated image with a low resolution, the size of pixels with a camera-shake-induced motion blur is also small. As a result, in the deteriorated image with a low resolution, the size of the blur kernel becomes small and the complexity of computations to do to infer the blur kernel is also little. Moreover, if the blur kernel were inferred using a deteriorated image with a high resolution and a large number of pixels from the beginning, then the inference could converge to a local minimal representing a different blur kernel from the true one. According to the multi-scale inference scheme, however, the blur kernel can be inferred more accurately.
Meanwhile, Patent Document No. 1 discloses a method for obtaining a less blurry restored image based on two pictures that have been gotten consecutively. According to that method, a blur kernel is inferred from each of a first image and a second image that has been gotten through a shorter exposure process than the first image. A synthesized blur kernel obtained by synthesizing together the two blur kernels inferred and a synthetic image obtained by synthesizing together the first and second images form a single restored image. According to this method, the restoration processing is carried out only if the first image is significantly blurred locally but is not carried out if the first image is blurred only slightly.